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(54) Abstract Title: Input or pointing device w Ith a camera 



(57) A pointing device having a moveable 3-D shape 
form 1 with bold visual features 2 is imaged by a 
television camera 3 to obtain a 2-D electronic 
image. The electronic image is evaluated by means 
of computer software in order to extract the visual 
features and measure their spatial dispositions 
within the 2-D image. This information enables six 
degrees of movement of the device to be 
determined. The movement so determined is 
applied to a 2-D graphical representation of a virtual 
3-D image displayed on a viewing screen of a 
computer monitor 5, or used to provide pointing 
information. The shape form 1 may be a cube, 
sphere (fig 4) or polyhedron (fig 3) with different 
coloured regions or LED's, or merely an assembly 
of LED's or the users hand; may be mounted on a 
base (figs 5-7); or enclosed by a transparent mouse 
type cover. A second camera (fig 8) may be 
provided for eclipsed or awkwardly lit features 2. 




00 

N> 

CO 
00 
00 

00 



The claims were filed later than the filing date but within the period prescribed by Rule 25(1) of the Patents Rules 1995. 
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Figure 8 
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Multi-vecfrnr pointing devir* 



The present invention is an extension of conventional forms of mouse in common 
usage with personal computers (PCs). PC mice, trackballs and touch pads provide 
two degrees of freedom while some joysticks or 'spaceballs' provide three or more 
degrees by incorporating a rotation as well as linear action. 

An objective of the present invention is to facilitate the representation of six degrees 
of freedom of movement of a 3-D object when displayed on a 2-D display screen. 
Six degrees of freedom are provided by three (x, y and z axes) translation directions 
and three rotational movements about those axes. 

Ttie closest prior art are so called "spaceballs" (for example from IBM), which are 
commercially available, and are able to provide up to six degrees of freedom. These 
are expensive and unwieldy to use. In use, it is difficult to maintain independent 
control of the degrees of freedom, using a "spacebar It is an objective of the 
present invention to overcome these difficulties. 

Conventional PC mice, while only providing two degrees of freedom, are also prior 
art for the present invention. 

It is further an objective of the present invention to facilitate the representation of 
six degrees of freedom of movement of a 3-D object, when displayed on a 2-D 
display screen, using existing low cost hardware. 

PC mice are used with solid computer modelling software to effect on-screen 
movement of 2-D pictorial representation of 3-D computer generated designs. 
Visual 2-D representation of all six degrees of freedom is difficult to portray using a 
2-D device such as a computer mouse. 
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It is an objective of the present Invention to provide up to six substantially 
independent degrees of freedom, which can be translated into corresponding 
degrees of freedom in a 2-D representation of a 3-D image, in a readily useable 
ergonomic form. 

In the present invention, a requisite number of independent degrees of freedom are 
derived from a 2-D optical image of a 3-D shape form. This is effected by converting 
the optical image into electronic form by existing known means, such as a charge 
coupled device (CCD) camera, and evaluating the electronic image in a way to 
determine movement in any of the six degrees of freedom. The present invention 
provides for any of the derived degrees of freedom to be appropriately applied to a 
virtual 3-D image displayed on a 2-D display, for example on a PC monitor. This 
enables an operator to manipulate the movements of a displayed image in a precise 
manner by virtue of the normal biofeedback loop translated through the operator. 

The evaluation of the electronic image is performed by appropriate computer 
software, which, when written is inexpensive to replicate and may be installed on 
any PC. 

According to the present invention, there is provided a multi-vector pointing device 
comprising a moveable shape form, a camera for producing a 2-D electronic image 
of the shape form in real time, characterised in providing pointing or viewing 
information of the object for software applications. 

According to the present invention, there is provided a multi-vector pointing device 
comprising a moveable three dimensional (3-D) shape form, a camera for producing 
a two dimensional (2-D) electronic image of the 3-D shape form in real time, 
characterised in providing evaluating means for determining up to six degrees of 
freedom of the 3-D shape form from the 2-D electronic image of the 3-D shape form 
and appropriately applying the degrees of freedom to a 2-D representation of a 
general virtual 3-D image displayed on a viewing screen. 
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Within the context of the present invention, the term "shape form" is used to 
describe an imagabie muitivector pointing device represented in three spatial 
dimensions. The shape form may also have two dimensional form, such as when 
constructed from a planar material such as sheet material, but would still occupy 
three dimensional space. 



Central to the invention is the use of a television camera, typically using CCD 
technology, for providing a 2-0 image of a 3-D shape form, in electronic form. The 
3-D shape form provides visual features whose positions are determined from the 2- 
D electronic form of the image. There are provided a sufficient number of visual 
features to determine the required number of degrees of freedom: for example three 
are require to produce six degrees of freedom, while two degrees of freedom may 
be determined from a single point feature. 

Evaluation of the positions of visual features in a 2-D electronic form of the image is 
performed in a prescribed way according to rules contained in computer software. 
This determines the directions and magnitudes of three orthogonal translations and 
three orthogonal rotations of the 3-D shape form. 

The visual features provided on the 3-0 shape form (forming part of the 3-D shape 
form) can be made distinctive, for example with coloured spherical surfaces, 
positioned in a suitable geometrical arrangement Coloured spherical surfaces may 
be positioned at the comers corresponding to the comers of a cube. It is within the 
scope of the present invention to provide a 3-D shape form having distinctive visual 
features disposed in any geometrical or non-geometrical 3-D configuration, suitable 
for facilitating their evaluation. Any particular configuration of visual features 
constitutes part of a 3-D shape form. 

Any means of providing visually distinct features lies within the scope of the present 
invention. This includes: polished or matt balls (spherical surfaces) illuminated by 
external light source; and small emitting regions such as light emitting diodes 
(l.e.d.s); small white, coloured or reflective stickers placed on the surface of a shape 
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form. The distinctive visual features will normally be attached to a featureless 
handheld rigid body, painted in one colour different from any of the visual features. 
For example, the body may be matt white or black, or any colour which is 
conveniently separated by electronic means in the 2-D electronic image. 

It is also within the scope of the present invention to include visual features on a 3- 
D shape form that are present by virtue of the geometry of the shape form, and 
which are highlighted by any given mode of illumination, such as room lighting. 

It is convenient that the distances separating pairs of visual features are pre- 
determined for use by the evaluating algorithms in the software. This is especially 
appropriate when the visual features are disposed according to a distinctive 
geometrical shape such as the comers of a cube. This, however, is not a 
prerequisite, and the necessary spatial information about the visual features may be 
determined using software based teaching procedures known per se. This is 
especially useful when the 3-D shape form has arbitrary shape with arbitrary visual 
features identifiable from a 2-D image of the shape form. 

While the 3-D shape form is primarily intended to be a handheld device, which 
replaces a computer mouse, it may be designed for use by disabled persons. It could 
for example be constructed from light weight materials, such as table tennis balls, 
which may be manipulated by a mouth held probe. 

It is within the scope of the present invention to include a user's hand or clenched 
fist as a 3-D shape form. Paper stickers may be attached to an operator's hand to 
provide easily identifiable visual features. 

It is also within the scope of the invention to provide evaluation software which will 
determine any of the six degrees of movement freedom of an arbitrary 3-D shape 
form (such as a clenched fist), by automatically identifying and tracking prominent 
visual features present in the shape form. This relies on measuring different 
amounts of displacement between different visual feature pairs, detected in the 2-D 
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electronic image. In this embodiment it is prudent to incorporate redundant 
information (redundant visual features) not only to improve accuracy, but also to 
cope with features coming in and out of prominence as the 3-D shape form rotates. 

Standard imaging processing techniques, known in the art, may be used in 
identifying visual features for the purpose of evaluation. By way of example, one 
method of evaluation involves computing the 2-D coordinates of centroids of 
respective visual features. Dimensions between all pairs of respective centroids are 
calculated, and variations in these values are used to determine movement in all six 
degrees of freedom. Many image processing and pattern recognition algorithms are 
known in the art, for the purpose of feature extraction, many of which are applicable 
to the present invention. For example edge detection may be used to identify dosed 
regions. For the purpose of this invention, a convenient number of dosed regions 
may be selected and their centroids computed. 

Tracking algorithms may be employed which correctly identify visual features as 
their shapes change with movement Previously documented processing techniques 
may be applied in this invention for the purpose of achieving the desired result 

Movement in any of the six degrees of freedom of the 3-D shape form is computed 
solely from distance-vector measurements made between visual features in the 2-D 
image and rates at which these distance-vectors change. Computation is simplified if 
actual distances between features is known in advance, but this is not a prerequisite 
for successful operation. 

It is also within the scope of the invention to provide more than one camera for 
viewing a shape form. This provides redundancy of information useful in 
circumstances when a feature is eclipsed from view by the shape form, or when 
lighting favours one viewing direction over another. 

The invention is described in detail with reference to the following figures: 
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Figure 1 shows salient features of a preferred embodiment of the invention. 

Figures 2 shows a shape form in the shape of a cube. 

Figure 3 shows a shape form in the shape of a regular solid polygon. 

Figure 4 shows a shape form in the shape of a sub-divided sphere. 

Figure 5 shows a shape form attached to a base via a ball joint 

Figure 6 shows a shape form attached to a base via coiled spring. 

Figure 7 shows a shape form attached to a base via an extendable stalk and ball 
joint 

Figure 8 shows an embodiment of the invention using two television cameras. 

With reference to figure 1: shape form, 1, supporting at least one visual feature, 2, 
is imaged by television camera, 3, to produce a 2-0 electronic image. This electronic 
image passes into PC, 4, for evaluation by image evaluating software. The image 
passes into the PC by known means, for example via a universal serial bus (USB) 
port. The television camera may conveniently be a "webcam" type. 

Monitor, 5, displays a 3-D representation of any object 6, which has been created or 
forms part of any software application using 3-D representation. 

The 3-D representation of object, 6, is moved on the monitor screen in any of the 6 
apparent degrees of freedom attributed to a three dimensional object, by imparting 
appropriate corresponding movement to shape form, 2. It will be appreciated that 
the 3-D representation of object 6, is actually a 2-D image on a flat screen. 
Movement in any of six degrees of freedom are therefore apparent movement in the 
3-D representation which enhance the 3-D perception of the object 
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Figure 2 shows a shape form with bold visual features, 1, positioned at the corners 
of a cube. In this case the visual features are partial spherical surfaces which may 
be polished metal to reflect external light sources, or else coated black, white or 
different colours, in matt or any other suitable surface finish. The features,l, 
positioned at the comers of cube, 2, may equally be emitting light sources such as 
light emitting diodes (l.e.d.s). Different coloured l.e.ds can be used to facilitate 
feature detection and identification. 

Figure 3 shows a regular 12 sided shape form having pentagonal faces. The faces 
themselves may be coloured to provide visual features or features may be attached 
to the comers as described above with reference to figure 2. The present invention 
is not restricted to this particular multifaceted shape form and any similar, not 
necessarily regular, multifaceted object may be used in accordance with the present 
invention. 



Figure 4 shows a spherical shape form, subdivided into distinct regions. The regions 
provide visual features, which may be coloured to facilitate identification. 

Figure 5 shows a shape form, mounted to a base, 7, via a ball-joint, 8. This allows 
the shape form to move in two horizontal dimensions as well as rotation about three 
perpendicular axes. The shape form is normally moved by hand, and a handle may 
be provided to assist hand movement 

Figure 6 shows a shape form with movement in all six degrees of freedom. The 
shape form, 2, is attached to a base, 7, via a loose coil, 9, which allows the shape 
form to be moved vertically as well as rotated about three orthogonal axes, relative 
to the base. The base is moveable in two horizontal axes; also the shape form is 
moveable in two horizontal axes relative to the base. 
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Figure 7 shows another way in which the shape form is moved in six degrees of 
freedom. The shape form is attached to a base, 7, by an extendable stalk,10, via 
pivots or a bail joint, 8. The base is moveable in two horizontal axes; also the shape 
form is moveable in two horizontal axes relative to the base. 

Any shape form may be attached to a base in accordance with figures 5, 6, and 7 or 
by other means. 

Figure 8 shows an embodiment of the invention incorporating more than one 
television camera. This provides redundant information, which is needed in cases 
where visual features become indistinct, such as when the shape form is an arbitrary 
shape. 

Any appropriate arbitrarily shaped object can be used as a shape form in accordance 
with the present invention. This includes an assembly of l.e.d.s without a regular 
geometrical relationship between them. 

The present invention includes a visual feature set and/or shape form, enclosed by a 
transparent mouse, type, cover. This provides aesthetic appeal, and the cover may 
be fashioned as a handle for ease of use. The cover can, for example, be dome 
shaped moulded transparent plastic. 

Included within the scope of the present invention is a shape form attached to a 
base via any semi-flexible coupling material, which allows complete freedom of 
movement of the shape form, but remains in a fixed position when released. 

Those skilled in the art will appreciate that while the present invention can return 
information about six degrees of freedom it could also be used to return less 
information, and therefore replace joysticks (normally three degrees of freedom) and 
mice (normally two degrees of freedom). 
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Multi-vector pointing device 



CLAIMS: 

1 . According to the invention, there is provided a multi-vector pointing 
device comprising a moveable shape form, a camera for producing 
a 2-0 electronic image of the shape form in real time, characterised 
in providing pointing or viewing information for software 
applications. 

2. According to the invention, there is provided a multi-vector pointing 
device comprising a moveable three dimensional (3-D) shape form, 
a camera for producing a two dimensional (2-D) electronic image of 
the 3-D shape form in real time, characterised in providing 
evaluating means for determining up to six degrees of freedom of 
the 3-D shape form from the 2-D electronic image of the 3-D shape 
form and appropriately applying the degrees of freedom to a 2-D 
representation of a general virtual 3-D image displayed on a 
viewing screen. 

3. A multi-vector pointing device according to claims 1 or 2 wherein 
the moveable shape form supports and provides visual features. 

4. A multi-vector pointing device according to claim 3 wherein the 
number of supported visual features is at least three. 

5. A multi-vector pointing device according to daims 1 or 2 and 3 
wherein the number of supported visual features is sufficient for at 
least three of the visual features to be distinct in the 2-D electronic 
image of the shape form at all times during operation. 



6. A mufti-vector pointing device according to claim 5 wherein visual 
features are rendered distinctive by virtue of visual features being 
of different colours. 

7. A mufti-vector pointing device according to claims 5 or 6 wherein 
visual features are disposed in any suitable geometrical 3-0 
configuration. 

8. A mufti-vector pointing device according to any of the claims 5, 6 or 
7 wherein the visual features are rendered distinctive by virtue of 
being individual spherical surfaces having polished or matt surfaces 
and which are illuminated by an external light source. 

9. A multi-vector pointing device according to any of the daims 5, 6 or 
7 wherein the visual features are rendered distinctive by virtue of 
being individual light emitters. 

10. A multi-vector pointing device according to claim 9 wherein 
individual light emitters are light emitting diodes. 

11. A multi-vector pointing device according to any one of claims 3, 4, 
5 or 7 wherein visual features are white, coloured or reflective 
stickers placed on the surface of a shape form. 

1 2. A multi-vector pointing device according to any of the previous 
claim wherein the shape form is a handheld rigid body. 

13. A multi-vector pointing device according to daim 12 wherein the 
shape form is a 3-D polyhedron. 

14. A mufti-vector pointing device according to claim 13 wherein the 
facets of 3-D polyhedron are visual features. 
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